We present a differentiable formulation of rigid-body contact dynamics for objects and robots represented as compositions of convex primitives. Existing optimization-based approaches simulating contact between convex primitives rely on a bilevel formulation that separates collision detection and contact simulation. These approaches are unreliable in realistic contact simulation scenarios because isolating the collision detection problem introduces contact location non-uniqueness. Our approach combines contact simulation and collision detection into a unified single-level optimization problem. This disambiguates the collision detection problem in a physics-informed manner. Compared to previous differentiable simulation approaches, our formulation features improved simulation robustness and a reduction in computational complexity by more than an order of magnitude. We illustrate the contact and collision differentiability on a robotic manipulation task requiring optimization-through-contact. We provide a numerically efficient implementation of our formulation in the Julia language called Silico.jl.

游戏理论运动计划者是控制多个高度交互式机器人系统的有效解决方案。大多数现有的游戏理论规划师不切实际地假设所有代理都可以使用先验的目标功能知识。为了解决这个问题，我们提出了一个容忍度的退缩水平游戏理论运动计划者，该计划者利用了与意图假设的可能性相互交流。具体而言，机器人传达其目标函数以结合意图。离散的贝叶斯过滤器旨在根据观察到的轨迹与传达意图的轨迹之间的差异来实时推断目标。在仿真中，我们考虑了三种安全至关重要的自主驾驶场景，即超车，车道交叉和交叉点，以证明我们计划者在存在通信网络中存在错误的传输情况下利用替代意图假设来产生安全轨迹的能力。

神经辐射场（NERF）已成功用于场景表示。最近的工作还使用基于NERF的环境表示形式开发了机器人导航和操纵系统。由于对象定位是许多机器人应用的基础，因此进一步释放了机器人系统中NERF的潜力，我们研究了NERF场景中的对象定位。我们提出了一个基于变压器的框架NERF-LOC，以在NERF场景中提取3D边界对象框。 Nerf-Loc将预先训练的NERF模型和相机视图作为输入，并产生标记为3D边界对象的框作为输出。具体来说，我们设计了一对平行的变压器编码器分支，即粗流和细流，以编码目标对象的上下文和详细信息。然后将编码的功能与注意层融合在一起，以减轻准确对象定位的歧义。我们已经将我们的方法与基于传统变压器的方法进行了比较，我们的方法可以实现更好的性能。此外，我们还提出了第一个基于NERF样品的对象定位基准Nerflocbench。

我们提出了一种算法，以（i）在线学习具有带有激光雷达的机器人的深度签名距离功能（SDF），以代表3D环境几何形状，以及（ii）鉴于此深度学习的地图，（ii）计划无碰撞轨迹。我们的算法采用了传入的激光扫描，并不断优化神经网络，以代表其当前附近环境的SDF。当SDF网络质量饱和时，我们将缓存网络的副本，以及学习的置信度指标，并初始化新的SDF网络以继续映射环境的新区域。然后，我们通过信心加权的计划来串联所有缓存的本地SDF，以提供全球SDF进行计划。为了计划，我们使用顺序凸模型预测控制（MPC）算法。 MPC规划师优化了机器人动态可行的轨迹，同时没有与全局SDF中映射的障碍物相撞。我们表明，与现有在线SDF培训的现有方法相比，我们的在线映射算法产生的地图更高。在Webots Simulator中，我们进一步展示了在线运行的组合映射器和计划者 - 自动导航，并且在未知环境中没有碰撞。

连接和自动驾驶汽车（CAVS）正在越来越广泛地部署，但是目前尚不清楚如何最好地部署智能基础架构以最大程度地发挥其功能。一个关键的挑战是确保骑士能够可靠地感知其他代理，尤其是被阻塞的药物。另一个挑战是，智能基础架构的渴望是自主的，并且很容易扩展到与现代交通信号灯相似的广阔部署。目前的工作提出了自我监督的交通顾问（SSTA），这是一种基础架构边缘设备概念，该概念利用与通信和共同培训框架共同利用自我监督的视频预测，以启用整个智能城市的自动预测流量。 SSTA是一款静态安装的摄像头，可俯瞰复杂的交通流量的交点或区域，可预测流量流作为未来的视频帧，并学会与相邻的SSTA进行通信，以在视野（FOV）中出现在视野中之前预测流量。拟议的框架旨在达到三个目标：（1）设备间的通信以实现高质量的预测，（2）对任意数量的设备的可伸缩性，以及（3）终身在线学习以确保对不断变化的环境的适应性。最后，SSTA可以直接广播其未来预测的视频框架，以供骑士进行自己的后期处理以进行控制。

我们提出了Dojo，这是一种用于机器人技术的可区分物理引擎，优先考虑稳定的模拟，准确的接触物理学以及相对于状态，动作和系统参数的可不同性。Dojo在低样本速率下实现稳定的模拟，并通过使用变异积分器来节省能量和动量。非线性互补性问题，具有用于摩擦的二阶锥体，模型硬接触，并使用自定义的Primal Dual内部点法可靠地解决。使用隐式功能定理利用内点方法的特殊属性，以有效计算通过接触事件提供有用信息的光滑梯度。我们展示了Dojo独特的模拟紧密接触能力，同时提供了许多示例，包括轨迹优化，强化学习和系统识别。

神经辐射场（NERF）最近被成为自然，复杂3D场景的代表的强大范例。 NERFS表示神经网络中的连续体积密度和RGB值，并通过射线跟踪从看不见的相机观点生成照片逼真图像。我们提出了一种算法，用于通过仅使用用于本地化的板载RGB相机表示为NERF的3D环境导航机器人。我们假设现场的NERF已经预先训练了离线，机器人的目标是通过NERF中的未占用空间导航到目标姿势。我们介绍了一种轨迹优化算法，其避免了基于NERF中的高密度区域的碰撞，其基于差分平整度的离散时间版本，其可用于约束机器人的完整姿势和控制输入。我们还介绍了基于优化的过滤方法，以估计单位的RGB相机中的NERF中机器人的6dof姿势和速度。我们将轨迹策划器与在线重新循环中的姿势过滤器相结合，以提供基于视觉的机器人导航管道。我们使用丛林健身房环境，教堂内部和巨石阵线导航的四轮车机器人，使用RGB相机展示仿真结果。我们还展示了通过教会导航的全向地面机器人，要求它重新定位以缩小差距。这项工作的视频可以在https://mikh3x4.github.io/nerf-navigation/找到。

Using Structural Health Monitoring (SHM) systems with extensive sensing arrangements on every civil structure can be costly and impractical. Various concepts have been introduced to alleviate such difficulties, such as Population-based SHM (PBSHM). Nevertheless, the studies presented in the literature do not adequately address the challenge of accessing the information on different structural states (conditions) of dissimilar civil structures. The study herein introduces a novel framework named Structural State Translation (SST), which aims to estimate the response data of different civil structures based on the information obtained from a dissimilar structure. SST can be defined as Translating a state of one civil structure to another state after discovering and learning the domain-invariant representation in the source domains of a dissimilar civil structure. SST employs a Domain-Generalized Cycle-Generative (DGCG) model to learn the domain-invariant representation in the acceleration datasets obtained from a numeric bridge structure that is in two different structural conditions. In other words, the model is tested on three dissimilar numeric bridge models to translate their structural conditions. The evaluation results of SST via Mean Magnitude-Squared Coherence (MMSC) and modal identifiers showed that the translated bridge states (synthetic states) are significantly similar to the real ones. As such, the minimum and maximum average MMSC values of real and translated bridge states are 91.2% and 97.1%, the minimum and the maximum difference in natural frequencies are 5.71% and 0%, and the minimum and maximum Modal Assurance Criterion (MAC) values are 0.998 and 0.870. This study is critical for data scarcity and PBSHM, as it demonstrates that it is possible to obtain data from structures while the structure is actually in a different condition or state.

Due to the high activation sparsity and use of accumulates (AC) instead of expensive multiply-and-accumulates (MAC), neuromorphic spiking neural networks (SNNs) have emerged as a promising low-power alternative to traditional DNNs for several computer vision (CV) applications. However, most existing SNNs require multiple time steps for acceptable inference accuracy, hindering real-time deployment and increasing spiking activity and, consequently, energy consumption. Recent works proposed direct encoding that directly feeds the analog pixel values in the first layer of the SNN in order to significantly reduce the number of time steps. Although the overhead for the first layer MACs with direct encoding is negligible for deep SNNs and the CV processing is efficient using SNNs, the data transfer between the image sensors and the downstream processing costs significant bandwidth and may dominate the total energy. To mitigate this concern, we propose an in-sensor computing hardware-software co-design framework for SNNs targeting image recognition tasks. Our approach reduces the bandwidth between sensing and processing by 12-96x and the resulting total energy by 2.32x compared to traditional CV processing, with a 3.8% reduction in accuracy on ImageNet.

We present a simple yet effective end-to-end Video-language Pre-training (VidLP) framework, Masked Contrastive Video-language Pretraining (MAC), for video-text retrieval tasks. Our MAC aims to reduce video representation's spatial and temporal redundancy in the VidLP model by a mask sampling mechanism to improve pre-training efficiency. Comparing conventional temporal sparse sampling, we propose to randomly mask a high ratio of spatial regions and only feed visible regions into the encoder as sparse spatial sampling. Similarly, we adopt the mask sampling technique for text inputs for consistency. Instead of blindly applying the mask-then-prediction paradigm from MAE, we propose a masked-then-alignment paradigm for efficient video-text alignment. The motivation is that video-text retrieval tasks rely on high-level alignment rather than low-level reconstruction, and multimodal alignment with masked modeling encourages the model to learn a robust and general multimodal representation from incomplete and unstable inputs. Coupling these designs enables efficient end-to-end pre-training: reduce FLOPs (60% off), accelerate pre-training (by 3x), and improve performance. Our MAC achieves state-of-the-art results on various video-text retrieval datasets, including MSR-VTT, DiDeMo, and ActivityNet. Our approach is omnivorous to input modalities. With minimal modifications, we achieve competitive results on image-text retrieval tasks.